S-DFP: shifted dynamic fixed point for quantized deep neural network training

Abstract Recent advances in deep neural networks have achieved higher accuracy with more complex models. Nevertheless, they require much longer training time. To reduce the time, methods using quantized weight, activation, and gradient been proposed. Neural network calculation by integer format improves energy efficiency of hardware for learning Therefore, fixed point However, narrow data representation range degrades accuracy. In this work, we propose a new named shifted dynamic (S-DFP) to prevent degradation training. S-DFP can change adding bias exponent. We evaluated effectiveness on ImageNet task ResNet-34, ResNet-50, ResNet-101 ResNet-152. For example, ResNet-152 is improved from 76.6% conventional 8-bit DFP 77.6% S-DFP.